feat: improve memory usage of zstd encoder by using our own pool management #2375
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently a single zstd encoder with default concurrency is used. Default concurrency causes EncodeAll to create one encoder state per GOMAXPROC, per default per core. On high core machined (32+) and high compression levels this leads to 1GB memory consumption per ~32 cores. A 1GB encoder is pretty expensive compared to the 1MB payloads usually sent to kafka. The new approach limits the encoder to a single core but allows dynamic allocation of additional encoders if no encoder is available. Encoders are returned after use, thus allowing for reuse. A benchmark emulating a 96 core system shows the effectiveness of the change. Previous result: ``` goos: linux goarch: amd64 pkg: github.com/Shopify/sarama cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkZstdMemoryConsumption-8 2 834830801 ns/op 3664055292 B/op 4710 allocs/op PASS ok github.com/Shopify/sarama 2.181s ``` Current result: ``` goos: linux goarch: amd64 pkg: github.com/Shopify/sarama cpu: 11th Gen Intel(R) Core(TM) i5-1135G7 @ 2.40GHz BenchmarkZstdMemoryConsumption-8 5 222605954 ns/op 38960185 B/op 814 allocs/op PASS ok github.com/Shopify/sarama 3.045s ``` ``` BenchmarkZstdMemoryConsumption-8 2 834830801 ns/op 3664055292 B/op 4710 allocs/op BenchmarkZstdMemoryConsumption-8 5 222605954 ns/op 38960185 B/op 814 allocs/op ``` A ~4x improvement on total runtime and a 96x improvemenet on memory usage for the first 2x96 messages. This patch will as a downside increase how often new encoders are created on the fly and the maximum number of encoders might be even higher.
|
CLA signed |
This was referenced Nov 11, 2022
dnwe
changed the title
dnwe
approved these changes
Nov 14, 2022
There was a problem hiding this comment.
Thanks for working on this and sharing benchmarks — this change looks good to me
cc @lizthegrey
Ooh, thanks. This probably will let us better bin-pack our telemetry ingest processes with other more memory-intensive workloads. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently a single zstd encoder with default concurrency is used. Default concurrency causes EncodeAll to create one encoder state per GOMAXPROC - per default per core.
On high core machined (32+) and high compression levels (32MB / state) this leads to 1GB memory consumption per ~32 cores. A 1GB encoder is pretty expensive compared to the 1MB payloads usually sent to kafka.
The new approach limits the encoder to a single core but allows dynamic allocation of additional encoders if no encoder is available. Encoders are returned after use, thus allowing for reuse, with a limit of 1 spare encoder to limit memory overhead.
A benchmark emulating a 96 core system shows the memory effectiveness of the change.
Previous result:
Current result:
A ~4x improvement on total runtime and a 96x improvemenet on memory usage for the first 2x96 messages.
This patch will as a downside increase how often new encoders are created on the fly and the maximum number of encoders might be even higher - however it should be in line with the actual used cores instead of the theoretical available cores.